Presentation: Tweet"Old and New Building Blocks Come Together For Big Data"

It has become a widely accepted truism that big data means that you have to throw away existing code and re-implement your algorithms using new file systems (like HDFS), new data stores (like HBase or Cassandra) and new programming paradigms (like map-reduce). It is said that this is necessary in order to achieve high performance and scalability.

I think that this need not be so. In fact, I think it should not be so.

I will demonstrate with concrete examples that old and new code can work together in big data environments with benefits on both sides. These concrete examples will use well-known components including Solr/Lucene, Storm, node.js and d3.js together with big data systems based on Hadoop to provide state-of-the-art recommendation systems. Moreover, the combination of these parts into one cohesive system shows just how important it is to avoid trying to make one kind of system do all kinds of jobs. Instead, by building a hybrid system, we can have the benefits of multiple specialist systems.